17. Exercise: Define a Model w/ Specifications

Model Design

Now that you've seen how to tune and balance a LinearLearner , it is your turn to put together all that you've learned and build a new model, based on a real, business problem. This exercise is meant to be more open-ended, so that you get practice with the steps involved in designing a model and deploying it. In this exercise you'll:

  • Create a LinearLearner model, according to specifications
  • Train and deploy the model
  • Evaluate the results
  • Delete the endpoint (after evaluation)

Here is what you'll see in the main exercise notebook:

EXERCISE: Train and deploy a LinearLearner with appropriate hyperparameters, according to the given scenario

Scenario:

A bank has asked you to build a model that optimizes for a good user experience; users should only ever have up to about 15% of their valid transactions flagged as fraudulent.

This requires that you make a design decision: Given the above scenario, what metric (and value) should you aim for during training?

You may assume that performance on a training set will be within about 5-10% of the performance on a test set. For example, if you get 80% on a training set, you can assume that you'll get between about 70-90% accuracy on a test set.

Your final model should account for class imbalance and be appropriately tuned. 

%%time
# instantiate and train a LinearLearner

# include params for tuning for higher precision
# *and* account for class imbalance in training data
%%time 
# deploy and evaluate a predictor
## IMPORTANT
# delete the predictor endpoint after evaluation

In this case, I will not be walking through a detailed solution (and there are multiple ways to approach this task and come up with a solution), but you can see one example solution in the solution notebook and on the next page.

Final Cleanup!

After completing these tasks, double check that you have deleted all your endpoints, and associated files. I'd also suggest manually deleting your S3 bucket, models, and endpoint configurations directly from your AWS console. You can find thorough cleanup instructions, in the documentation .

Next Concept